Homework 07 - Solution
Due Thursday, 11/18/2021, 2:00 PM
Instructor: Prof. Seungchul Lee
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH
For your handwritten solution, scan or take a picture of them (you can write it in markdown if you want).
For your code, only .ipynb file will be graded.
- Please write your NAME and student ID on your .ipynb files. ex) IljeokKim_20202467_HW07.ipynb
Please compress all the files to make a single .zip file
- Please write your NAME and student ID on your .zip files. ex) DogyeomPark_20202467_HW07.zip
- Submit it to PLMS
Do not submit a printed version of your code. It will not be graded.
(1) Design your train, validation, and test dataset using each folder.¶
In [1]:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
import cv2
import random
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Activation, Dense, Flatten, BatchNormalization, Conv2D, MaxPool2D, GlobalAveragePooling2D
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.metrics import categorical_crossentropy
import warnings
warnings.filterwarnings(action='ignore')
%matplotlib inline
In [2]:
train_path = './data_files/train'
validation_path = './data_files/validation'
test_path = './data_files/test'
In [3]:
classes=['berry', 'bird', 'dog', 'flower']
train_batches = ImageDataGenerator(rescale = 1./255,) \
.flow_from_directory(directory=train_path, target_size=(224,224), classes=classes, batch_size=256)
validation_batches = ImageDataGenerator(rescale = 1./255,) \
.flow_from_directory(directory=validation_path, target_size=(224,224), classes=classes, batch_size=256, shuffle=False)
test_batches = ImageDataGenerator(rescale = 1./255,) \
.flow_from_directory(directory=test_path, target_size=(224,224), classes=classes, batch_size=256, shuffle=False)
(2) Plot 10 random data.¶
In [4]:
imgs, labels = next(train_batches)
def plotImages(images_arr):
fig, axes = plt.subplots(1, 10, figsize=(20,20))
axes = axes.flatten()
for img, ax in zip(images_arr, axes):
ax.imshow(img)
ax.axis('off')
plt.tight_layout()
plt.show()
In [5]:
plotImages(imgs)
Problem 2: Transfer Learning¶
We will use VGG16 architecture for training our dataset. As you can see below image, VGG16 architecture has 16 layer blcoks and a lot of training parameters. Because machine learning libraries, such as Tensorflow, Keras, and Pytorch, provide pre-trained models for ImageNet, we don't have to design and train the model from scratch.
(3) Design VGG16 model using machine learning libraries, such as Tensorflow, Keras, and Pytorch¶
In [6]:
vgg16_model = tf.keras.applications.vgg16.VGG16()
In [7]:
vgg16_model.summary()
(4) Modify original VGG16 architecture. As you can see below image, we will modify only fully connected layer part. Also, since we will use pre-trained parameters, parameters of the feature extraction part must be fixed.¶
In [8]:
model = Sequential()
for layer in vgg16_model.layers[:-4]:
model.add(layer)
In [9]:
for layer in model.layers:
layer.trainable = False
In [10]:
model.add(Conv2D(filters=1024, kernel_size=(3, 3), padding='same', activation='relu'))
model.add(GlobalAveragePooling2D())
model.add(Dense(units=4, activation='softmax'))
In [11]:
model.summary()
(5) Train the modified VGG16 model. You can get a perfect score if validation accuracy exceeds 90%.¶
In [12]:
model.compile(optimizer=Adam(learning_rate=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(x = train_batches,
steps_per_epoch = len(train_batches),
epochs = 5,
verbose = 1,
validation_data = validation_batches
)
Out[12]:
(6) Print your accuracy about test dataset.¶
In [13]:
test_imgs, test_labels = next(test_batches)
In [14]:
plotImages(test_imgs)
In [15]:
predictions = model.predict(x = test_batches, steps = len(test_batches), verbose = 0)
In [16]:
print('test accuracy: ', accuracy_score(y_true=test_batches.classes, y_pred=np.argmax(predictions, axis=-1)))
Problem 3: Class Activation Maps¶
(7) As shown in the given figure, plot the CAM results.¶
In [17]:
get_output = tf.keras.backend.function([model.layers[0].input],
[model.layers[-3].output, model.layers[-1].output])
conv_outputs, predictionss = [], []
c, p = get_output([test_imgs])
class_weights = model.layers[-1].get_weights()[0]
In [18]:
output = []
for num, idx in enumerate(np.argmax(p, axis=1)):
cam = tf.matmul(np.expand_dims(class_weights[:,idx], axis = 0),
np.transpose(np.reshape(c[num], (7*7,1024))))
cam = tf.keras.backend.eval(cam)
cam = np.reshape(cam, (7,7))
cam = (cam-np.min(cam))/(np.max(cam)-np.min(cam))
cam = np.expand_dims(np.uint8(255*cam), axis=2)
cam = cv2.applyColorMap(cv2.resize(cam, (224, 224)), cv2.COLORMAP_JET)
cam = cv2.cvtColor(cam, cv2.COLOR_BGR2RGB)
output.append(cam)
In [19]:
idx_list = [random.randint(0,255) for r in range(5)]
idx_list
for i in idx_list:
plt.figure(figsize=(6,3))
plt.subplot(121)
plt.imshow(test_imgs[i])
plt.title('True: {} / Pred: {}'.format(classes[np.where(test_labels[i] == 1)[0][0]], classes[np.argmax(p[i])]))
plt.axis('off')
plt.subplot(122)
plt.imshow(test_imgs[i])
plt.imshow(output[i], 'jet', alpha = 0.7)
plt.title('Class Activation Map')
plt.axis('off')
plt.tight_layout()
plt.show()
